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Agenda 


•  I  ntroduction  of  AEA I PT 

•  Process  I  mprovement  Objectives  for  FY09 

•  Customizing  processMax® 


■  processMax®  overview 

■  Customize  for  System  and  Software  Development  Project 

■  Customize  for  Data  Base  Development  Project  (EWDS) 

■  Customize  to  integrate  Lean  Six  Sigma  (LSS)  and  CMMI  high 
maturity  level  practices 


•  I  ntegrate  NAVAI R  Lean  Six  Sigma  into  AEA 
I  PT  critical  processes 


■  Quantitative  Defect  Management  (QDM) 

■  Quantitative  Requirements  Management  (QRM) 

■  Causal  Analysis  and  Resolution  (CAR) 


AEA  IPT 


UNCLASSIFIED  II  FOR  PUBLIC  RELEASE 


)  'J  r'  . 
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Organization  Structure 

Airborne  Electronic  Attack  I PT 


CORE  AEA 
Functional  Group  POC 


Intelligence  Operations 


EW  Sensor  Engineering 


Jammer  Techniques  Optimization 


EA  Systems  Engineering 


Software  Development 


Avionics  /  Sensors  Laboratories 


EA  Integrated  Test  &  Evaluation 


EW  Battle  Space  Management 


EA  Interference  Cancellation 


EA  Jammers 


Fleet  Help  Desk 


EA-18G  /  AEA 


Process  Improvement  Lead 
Contracting  Officer  Rep 
Training  Manager 


ACQUISITION  AND 
FLEET  SUPPORT 
Product  Teams 


EW  Database  Support  (EWDS) 


ETIRMS  UPC 


EA-6B  MPE  /  UPC 


EA-6B  Aircrew  Trainers 


Next  Generation  Jammer  Pod 


EA-18G  AEA  UPC 


E-2C/MH-60  HAWK  Tool 


Intrepid  Tiger  Pod 


Joint  EW  Effects  Lab 


N  A  V 
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Sponsor  Product  Alignment 
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AEA  Products  /  Services 


•  Electronic  Warfare  Database  Support  (EWDS) 

■  EOB  product  to  all  Navy  Aircraft  using  J  MPS 

>  EA-6B,  EA-18G,  F/A-18C/D,  F/A-18E/F,  MH-60,  E-2C,  AV-8B,  ... 

>  ETI  RMS  &  EWDS  to  Navy,  Air  Force,  NSA,  J  SF,  MMA  and  other  customers 

•  AEA  Mission  Planning 

■  EW  Tactical  I  nformation  and  Report  Management  System  (ETI  RMS) 
Unique  Planning  Component  (UPC)  for  EA-6B  &  EA-18G 

■  EA-18G  AEA  UPC 


■  EA-6B  Mission  Planning  Environment  (MPE)  +  MH-60/ E-2C  HAWK  Tool 

•  AEA  J  ammer  Techniques  Optimization  (J  ATO) 

•  EA-6B  I  CAP  1 1  and  I  CAP  1 1 1  Block  &  GWOT  Upgrades 

■  Software  Maintenance,  I  ntegration,  and  Test  (including  Aircrew  Trainers) 

■  Block  System  Upgrade  Design,  Development,  Integration  and  Test 

•  EA-18G  AEA  Block  Upgrades 


■  I  ncluding  AEA  Systems  Engineering  +  I  ntegrated  Test  &  Evaluation 


•  I  ntrepid  Tiger  Pod  Software  Support  Activity 


N 
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AEA I  FT  Team  Composition 


78  Support  Contractor 

•  Northrop  Grumman 

•  L-3  Corporation 

•  Wyle  Labs 

•  Digital  Wizards 

•  GBL,  JTI  &SIMSUM 


Personnel 


□  Government 
■  Contractor 


130+  Direct  Funded  Government  Employees 
2  Military  Officers  (Excluding  1  vacancy) 


Personnel  with  AEA  Expertise: 

-  Over  85%  Engineers 

-  AEA  On-site  System  Engineering  Expertise  is  Still  Largest  in  Nation 

-  Depth  of  AEA  Experience  averages  over  10  years  per  individual 

NAV/^ 


II  R 


AEA  IPT 


UNCLASSIFIED  II  FOR  PUBLIC  RELEASE 


6 


■1 

WMl 


Process  Status 


AEA  IPT  achieved  CMMI  Level  3  in 

June  2007 

j - 1= 
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FY  09  Process  Obj  ectives 


I  mprove  Performance  by  implementing 
Continuous  Process  I  mprovement  (CPI ) 

NAVAI R  Lean  Six  Sigma  ( LSS) 

per 

DoD  Directive  5010.42 


Kl  AM/ 
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DoD  Directive  5010.42 


•  Strengthen  joint  operational  Combatant  Command 
and  Military  Department  capabilities  including 
making  improvements  in: 

(1)  Productivity 

(2)  Performance  against  mission  (availability,  reliability,  cycle  time, 
investment,  and  operating  costs) 

(3)  Safety  -  Flexibility  to  meet  DoD  mission  needs  -  Energy  efficiency 

•  CPI  /  LSS  programs  shall  be  used  to  help  meet 
organizational  objectives 

■  CPI /LSS  methods,  terminology,  training  plans,  and  other 
program  elements  may  be  adapted  as  required 

■  Given  diverse  operational  requirements,  the  DoD  Components 
shall  have  full  flexibility  to  identify  CPI /LSS  focus  areas  and 
training  plans  and  may  adapt  other  CPI /LSS  program  elements 
for  their  use 
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AEA !  FT  Strategy  to 
Implement  DoDD  50X2.42  (X) 

•  Responsible  Parties 

■  AEA  I PT  Management  Team  and  Competency  Aligned 
Organization  (CAO) 

■  Product  Leads,  Project  Managers  and  Team  Members 

■  Process  Management  Team 

•  Define  and  align  AEA  I  PT  Performance  Objectives 
with  NSPS 

■  For  each  product  release: 

>  I  mprove  Cost  by  X% 

^  I  mprove  Schedule  by  X%  National  Security 

>  I  mprove  Quality  by  X%  Personnel  System 


1  *J  A.  ✓ 
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AEA  ( FT  Strategy  to 
I  mplement  DoDD  5012.42  (2) 


•  Ensure  consistent  Organizational  performance 

■  Customize  processMax®  to  integrate  Lean  Six  Sigma  (LSS)  tool 
sets  and  to  support  non-software  products  (EWDS,  J  ATO) 

■  I  ntegrate  LSS  Tool  Sets  into  Critical  Process  Activities 

>  Quantitative  Defect  Management  (QDM) 

>  Quantitative  Requirements  Management  (QRM) 

>  Earned  Value  Management  (EVM) 

>  Causal  Analysis  and  Resolution  (CAR) 

•  I  ntegrate  Al  RSpeed  LSS  Methodology  into  AEA  I  FT 
Culture 

■  Conduct  Lessons  Learned  to  evaluate  past  performance,  identify 
improvement  opportunities  and  implement  Organizational  Change 
Requests  (OCRs)  using  LSS  projects: 

>  Black  Belt,  Green  Belt,  etc. 
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Customizing  processMax® 


DOD-NAVAIR  Directed 


Continuous  Process 
Improvement  (CPI)  by 
Integrating  Lean  Six  Sigma 
into  critical  processes 

_ _ _ J 
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processMax®  Tool 
Overview 


•  A  web- based  project  management  software  tool 
used  for  project  and  organization  personnel  to 
follow  a  defined  process. 

■  Includes  all  processes,  procedures,  guidelines,  criteria, 
templates,  and  forms  used  by  the  organization 

■  Serves  as  a  document  repository  for  project  and  organizational 
work  products 

■  Provides  configuration  management  capabilities  that 
include  version  control,  change  control,  and  process  history 

■  Supports  project  management  activities  such  as  project 
planning,  tracking  of  Actions/ Issues,  Decisions,  Risks,  Role 
Assignments,  Defects,  Training  status,  etc. 

■  Provides  the  structure  to  ensure  that  a  standard  project 
process  is  followed  by  all  projects  and  allows  for  the  tailoring 
of  those  processes  as  needed 


AEA  IPT 


UNCLASSIFIED  II  FOR  PUBLIC  RELEASE 


13 


AEAIPT 

processMax®  Customization 

•  A  Decision  Analysis  and  Resolution  (DAR) 
process  activity  supported  the  decision  to 
customize  the  existing  CMM  processMax®  tool 

■  Critical  factors  in  this  decision  included: 

>  Pragma  Systems  delay  in  releasing  a  completed  CMMI  Level  3 
version  of  processMax® 

>  Concerns  that  the  new  release  might  not  align  with  AEA  I PT  best 
practices 

>  Costly  manual  transfer  of  project  data  to  the  new  version 

>  Modifications  could  be  made  quickly  by  AEA  Process 
Management  Team  to  existing  processMax®  interface  to  rapidly 
deploy  CMMI  across  the  Organization 

>  Projects  Team  would  not  have  to  learn  a  new  tool 

S  Training  efforts  could  be  concentrated  on  the  new  CMMI  Process 
Activities 


* 

J 
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AEA I PT  processMax® 
Customization  Examples  ( 1} 


Measurement  &  Analysis  ( M&A)  Process  Activity 


•  Simplify  pMax 

■  Helps  projects  by  facilitating 
collaboration  and  collection 
of  M&A  artifacts 

•  M&A  Artifact  repository 
contains: 

■  Meeting  agenda,  minutes 
and  measurement  indicators 


My  Work  I  Action  Itemslssues  I  Decisions  I Risks  ( Defer' .  I  Requests  I  Glossary  I  Utilities  [  Folder  I  D 


Added  new  Project 
Measurement  and 
Analysis  work 
product  and 
process  steps  to 
processMax® 


Control  Page 


Stand  .,d  AEA  IPT  Software  Project  Process  Version  1 .5 


0  Project  Measurement  and  Analysis  Plan 


■  Action  item  logs  and 
decisions 


Actions: 


New  ■  Examples  M  Templates  I  Report 

'-•per  I  p-r-ir/e  I 


Display: 


Workflow  M  Folders  M  Help 


Filter: 


View  I  Edit  M  Clear 


■  Electronic  approvals  of  M&A 
Plan 


Control 

Number 

Revision 

Author 

Status 

Status 

Effective 

Date 

Reviews  And 
Approvals 

SCCB  Approved 
Change  Request 
(s) 

None 

NAV 
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AEA I  FT  processMax® 
Customization  Examples  (2) 


Process 
developed  and 
aligned  with 
NAVAIR 
Engineering 
Guidelines 


Added  new 
process  steps 
to  implement 
AEA  IPT  best 
practices 


Modified  Process  Steps 


Process  H  PR  [  V  CMM  [  W  Help 


H  Senior  Manager  -  Graves,  Allan  E 
X  Product  Lead  -  Clarke,  Lynne  A 
A  Project  Manager  -  Bukowski,  Dan 
R  Configuration  Manager  -  Taylor,  Debra  A 
Standards  Compliance  Manager  -  Davis,  David 
R  Subcontract  Manager  -  Clarke,  Lynne  A 

Requirements  Engineering  Lead  -  Puckett,  Ken 


Establish  and  Maintain  the  Operational  Concepts  and  Scenarios 
Establish  and  Maintain  Functional  Requirements  Document  (FRD) 


Establish  and  Maintain  System  and  Subsystem  Specification  (SSS) 


Establish  and  Maintain  Interface  Requirements  Specification  (IRS) 

#*  Prepare  to  Analyze  Requirements 
#'  Analyze  Requirements 

Validate  Requirements  Using  Comprehence  Methods 
#*  Prepare  Requirements  Traceability  Matrix 

Prepare  Software  Requirements  Statisics  Report 
*  Design  Lead  -  Beylin,  Chris  - 

&  Programmers  -  Alquist,  James  G,  Andrews,  Ramona,  Beylin,  Chris,  Blanche] 
^  Testing  Lead  -  Pacleb,  Tony 
tL  Product  Documentation  Lead  -  Hudson,  Stephen  A 


Process  step 
description 
reflects  actual 
practices 


My  Work  Action  Items  /Issues  Decisions  Risks  Defects  Rer 


Requirements  Engineering  Lead  -  Puckett,  Ken 


ICAP  III  Block  3 


Establish  and  Maintain  System^!!  Subsystem  Specification  (SSS) 

Re  quire  d  Re  ading quire  dPersonnel  Outputs 

DESCRIPTION:  The  System/SJ^stem  Specification  (SSS)  specifies  the  requirements  for  a 
system  or  subsystem  and  the^rethods  to  be  used  to  ensure  that  each  requirement  has  been 
satisfied.  The  SSS  is  used^s  the  basis  for  project  design  and  qualification  testing.  It  is 
recommended  that  the  developed  using  the  format  and  contents  inDI-IPSC-81431. 

PROCESS  STEP 

Analyze  and  develop  a  System  /  Subsystem  Specification  (SSS):  The  Requirements  Lead 
shall  coordinate  with  relevant  stakeholders  to  analyze  and  transform  the  FRD  into  SSS 
requirements  that  meet  established  acceptance  criteria.  The  SSS  shall  be  developed  to  formalize 
customer  requirements,  such  that  they  clearly  communicate  desired  system/software  product 
attributes  to  the  developers. 

Verification:  The  Requirements  Lead  shall  schedule  a  technical  review  or  peer  review  of  the  SSS 
to  find  defects  that  require  corrective  action.  Any  significant  defects  found  shall  result  in  the 
return  of  the  SSS  to  the  [analyze  and  develop  a  SSS]  process  step. 

Validation:  The  Project  Manager  and  Requirements  Lead  will  conduct  Technical  Interchange 
Meetings  to  provide  assurance  that  the  SSS,  as  formally  presented,  accurately  reflects  stakeholder 
expectations  for  the  system  and  subsystem.  Each  of  the  stakeholders  must  verify  that  their  needs 
are  being  met  by  the  SSS.  System  Risk  are  identified  and  reconciled  among  the  stakeholders.  Any 
deviations  from  their  expectations  shall  be  negotiated  with  the  sponsor  and  Project  Manager. 

Obtain  commitment  and  approval:  On  concurrence  that  the  SSS  meets  mission  objectives,  as 
stated  by  the  stakeholders,  and  that  the  acceptance  criteria  have  been  met,  the  SSS  shall  be 


zi 


NAV 


AEA  IPT 


UNCLASSIFIED  II  FOR  PUBLIC  RELEASE 


A)  R 


16 


I  incorporation  of 
Lean  Six  Sigma  ( 1) 


processMax  5.0e  -  Dry  Run  -  Update  Form  -  Defect  -  3  -  Microsoft  Internet  Explorer  provi 


Set  Sev 

Severity:  * 

|  Critical 

J 

Number  of  Actual  Defects: 


Accurate  defect  data  capture  is  critical 
for  project  performance  using  the 
LSS  Measure  and  Analysis  Phase 


Select  Defect  Category  Required  Reading 


Requirement  Defect  Category  ' 


Design  Defect  Category 


Code  Defect  Category: ' 


Hold  the  Control  key  down  while  using  the  mouse  to  select  or  deselect  items. 


Correctness 

Completeness 

Consistency  d 

Hold  the  Control  key  down  while  using  the  mouse  to  select  or  deselect  items. 
Logic  3 

Input  I 

- 1 

Hold  the  Control  key  down  while  using  the  mouse  to  select  or  deselect  items. 


Logic 

Input 

Data  Handling 


Ij 


Quality  Characteristics  Affected: 


Discovered  Via:* 

Life  Cycle  Phase  Originated:  * 
Life  Cycle  Phase  Discovered:  * 


Select  Quality  Characteristics  Required  Reading 

Hold  the  Control  key  down  while  using  the  mouse  to  select  or  deselect  items. 

Functionality 

Functionality .... 

Reliability 
Usability 
Efficiency 
Maintainability 

|  Peer  Review 

|  Planning 


Defect  categories 
were  redefined  to 
accurately  reflect 
project  environment 
and  source  of  defects 


w 


AEA  IPT 


UNCLASSIFIED  II  FOR  PUBLIC  RELEASE 


HAY 


6  J  ¥ 


17 


I  ncorporation  of 
Lean  Six  Sigma  (2) 


LSS  tool  sets  and  procedures,  like  Fish  Bone  and  Five  Whys,  are  integrated 
into  processMax®  to  guide  users  in  performing  cause  and  effect  analysis 


As  defects  are  fixed,  improvement 
proposals  are  identified  and  LSS 
projects  are  initiated  via  OCRs 


m  * 

\  Cause  and  Defect  Analysis 

Required  Reading  | 

Root  Cause  De^ription:* 

- 3 

\ 

zl 

Improvement  Proposal  Type:  * 

Hold  the  Control  key  down  while  using  the  mouse 

to  select  or  deselect  items. 

Tools 

Methods 

_ i 

liCommunications 

■Bid 

Improvement  Proposal  Description: 

- 3 

* 

d 

AEA  IPT 


UNCLASSIFIED  II  FOR  PUBLIC  RELEASE 


18 


I  ncorporation  of 
Lean  Six  Sigma  (3) 


14  T 


12  - 


10 


8  -■ 


6  -■ 


DEFECT  PARETO 


n 


-+- 


ii 

E  F  A  G 


-+- 


-+- 


H 


H  I  J 

Defect  C.ms€j==j  Defect  Cum  Freq. 


Defect  data  can  be 
extracted  from 
processMax®  into  an 
Excel  file  to  perform 
cause  and  effect 
analysis 


Phase  Detected 

Phase  Originated 

Requireme 

Design 

Coding 

System 

OT/DT 

Total 

Requirements 

46 

46 

Design 

5 

19 

24 

Coding 

1 

18 

66 

85 

Software  Integration 

0 

0 

0 

0 

System  Testing 

1 

9 

54 

64 

OT/DT 

0 

0 

0 

0 

Total 

53 

46 

120 

219 
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Improve  Performance  with 
Quantitative  Defect  Management 


DoD-NAVAIR  Directed 
Continuous  Process 
Improvement  (CPI)  by 
Integrating  Lean  Six  Sigma 
into  critical  processes 
_ _ _ ) 


J 
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Quotations 


Quality  is  never  an  accident,  it  is  always  the  result  of  high  intention, 
intelligent  direction  and  skillful  execution;  it  represents  the  wise 
choice  of  many  alternatives/7  William  A.  Foster 


"If  we  are  busy  doing  rework  for  defects,  we're  not  innovating  AND 
we  are  costing  the  company  lots  of  money."  Anonymous 

"Finding  and  fixing  defects  accounts  for  much  of  the  cost  of  software 
development  and  maintenance."  -  Watts  S.  Humphrey 


"I  t  is  much  less  expensive  to  prevent  errors,  than  to  rework,  scrap, 

or  service  them."  Philip  Crosby 


A I  R 
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I  introduction 

Quantitative  Defect  Management  ( 1) 


30% 

25% 

20% 

15% 

10% 

5% 

0% 


Defect  Removal  by  Phase 


Requirements 

Reviews 


Design 

Reviews 


Code 

Reviews 


Formal 

Testng 


Remaining 

Detects 


Facilitate  gradual  shift  from 
“Fix-on-Failure”  to  Prevention 
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I  introduction 

Quantitative  Defect  Management  (2) 


ANALYSIS  CKHSlCrS  CODING  TESTING  IN  USE 


Table  1:  Time  to  Fix  Defect  That  Escapes  Stc$e  (in  hours) 


Requirement 

Design 

Coding 

Development 

Test 

Acceptance 

Test 

During 

Operation 

1 

3-6 

10 

15-40 

30-70 

40-1,000 

24  CrossTalk  The  Journal  of  Defense  Software  Engineering 


Defect  removal  effort  can  increase  by  10  times  for 

each  stage  it  goes  undetected 
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AEA I PT  Lessons  Learned 


•  AEA  I  PT  Best  Practices 

■  Test  processes  sufficiently  robust  to  detect  most  defects 

>  Quality  of  released  product  is  consistently  high  across  the  AEA  I  PT 


•  AEA  I  PT  I  improvement  Opportunities 

■  Need  to  improve  defect  detection  during  Requirements,  Design 
and  Code  phases 

>  Consistency  in  counting  defects,  in  capturing  effort  /  size  &  in  logging 
defects 
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Three  Al  RSpeed 
Black  Belt  Projects 


•  Three  Al  RSpeed  Black  Belt  Proj  ects  I  m proved  the 
Defect  Removal  Effectiveness  (DRE)  Process  for 
Software  I  ntensive  Products: 

■  Requirements  Development  Phase 

■  Design  Phase 

■  Code  &  Unit  Testing  Phase 

•  Quantitative  Defect  Management  Process  Goals 

■  Discover  and  remove  more  defects  earlier  in  the  development 
lifecycle  to  support  'On-time'  delivery  objectives 

■  Reduce  rework  efforts  to  improve  Cost  and  Schedule 

■  I  mprove  Quality  Performance 

■  Evolve  Defect  Detection  Model 
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Strategy  to  I  mplement 
Quantitative  Defect  Management 


•  I  mprove  and  Maintain  Defect  Prevention  Techniques 

■  Measure  the  Effectiveness  of  each  Peer  Review 

■  Statistically  Analyze 

>  Performance  of  each  Peer  Review 

>  Defect  removal  effectiveness  at  the  completion  of  each  phase 

•  I  ntroduce  Quantitative  Defect  Management  Method 

■  Statistically  Analyze  Project  performance  against  AEA  I PT 
Performance  baseline 

■  Predict  Quality  and  Cost  Performance  using  a  Defect  Detection 
Model  (DDM) 

•  I  ntroduce  Causal  Analysis  and  Resolution  Process 

■  Determine  Root  Causes,  take  Corrective  Actions  to  improve  quality 
and  prevent  reoccurrence 


*  j  i 

l  '  J  r 
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Quantitative  Defect  Management 
(AEA I  FT  7  Step  Process) 
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Establish  AEA I PT 
Performance  Baselines 


AEA  IPT  Quality  &  Process  Performance  Baseline 


Baseline  Last  Revised  On: 


Product  Quality: 


Defect  Density: 


Mean 

UCL 

LCL 

Unit 

Defect  Density  (All  Phases) 

36 

39 

33 

KLOC 

Residual  Defect  Density 

0.2 

KLOC 

Process  Performance: 

Defect  Removal  Effectiveness: 


Sub  Process  Perform 


Phase 

Mean 

UCL 

LCL 

Requirements 

50% 

53% 

48% 

Design 

50% 

53% 

48% 

Coding 

45% 

65% 

57% 

Peer  Review  Coverage: 


Sub  Process 

Required 

Tolerance 

Req.  Peer  Review 

100% 

+/- 10% 

Design  Peer  Review 

100% 

+/- 10% 

Code  Peer  Review 

50% 

+/- 10% 

Sub  Process 


Attributes 


Req.  Peer  Review 


Design  Peer 
Review 


Code  Peer  Review 


Defects/Unit  Size 


Defects/ Review  Time 


Review  Time/Unit  Size 


Defects/Unit  Size 


Defects/  Review  Time 


Review  Time/Unit  Size 


Defects/Unit  Size 


Defects/  Review  Time 


Review  Time/Unit  Size 


Mean 


UCL 


k  kl  Dmd  Ma  Om  D-irnlinn  f\kinrhwnr  Hhnninn  OK-ita  Donm  Onwimn  rtArinnOnwimif  rAllB 


•  Establish  performance 
baselines  for: 

Defect  Density  (all  phase) 
Residual  Defect  Density 
Defect  Distribution  (by  Phase) 

Defect  Removal  Effectiveness 

•  Requirement 

•  Design 

•  Code  &  UT 

Peer  Review  (by  Phase) 

•  Defects/Unit  Size 

•  Defects/Review  Time 

•  Review  Time/Unit  Size 


HI 


J*  .  J  R 
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Establish  Project  Objectives 


Define  Project  &  Quality  Performance  Objectives 

AEA IPT  Business  Goals  and  Objectives  AEA IPT  Quality  &  Process  Perfromance  Objectives  Customer  and  Other  Stakeholder's  Objective 


<Documentthe  Business  Goals  &  Objectives- Protectthis 
spa> 


Project  Objective 


<ltcan  be  same  as  Org.  objectives  or  PM  can  add  more  to 
meetthecustomerandotherstaheholder'sobjective> 


<Documentthe  Quality  &  Process  Performance  Objectives> 


Project  Goals: 


Description  of  Measure 

Mean 

USL 

L>1 

DRE- Requirements  Phase 

65% 

67% 

DRE-  Design  Phase 

62% 

65% 

DRE -Code  and  Unit  Test 

45% 

49% 

•Project  establishes  objectives 
based  on: 

•Customer  (PMA) 

•IPT  Objectives 
•Project  Past  Performance 

•Project  defines  Quantitative 
Objectives  for  each  phase  of 
Defect  Removal  Effectiveness 

•Requirements 
•Design 
•Code  &  UT 


H  A  V, 
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Establish  Quality  Control  Target 

Define  Project  Parameters  &  Estimates 


Name  of  the  Project  AEAIPTPI  Project  Size  Estimates  9.50  KLOC 

Name  of  Project  Manager  Tuan  Le 


Defects  Origination  Estimates 


Phase 

Mean 

UCL 

LCL 

Requirements 

86 

95 

76 

Design 

86 

91 

80 

Coding 

154 

159 

149 

Software  Integration 

10 

13 

7 

System  Testing 

7 

9 

4 

Total 

342 

368 

316 

Defect  Detection  -  Tar 

get 

Phase 

Mean 

UCL 

LCL 

Requirements 

56 

57 

54 

Design 

72 

74 

69 

Coding 

113 

118 

110 

Software  Integration 

10 

System  Testing 

57 

OT/DT 

34.2 

Defects  -  Origination  Estimate  Vs  Detection  Target 


Project  Manager  estimates  the 
target  number  of  defects  originated 
&  removed  by  phase  to  establish 
project  objectives 

Defect  estimation  model  will  be 
based  on  historical  data  and 
organizational  performance  baseline 


Requirements  Design 


Coding 


Software 

Integration 


System 

Testing 


Defects  Origination  Estimates 


I  Defect  Detection  -  Target 


N 


OT/DT 


I  R 
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Statistically  Manage  Peer 
Review  Performance 


Design  Phase  -  Peer  Review 


Phase  Originated 


— 

J  ™  - 

■1.  iu  — 
_ 
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Statistically  Manage  Defect  Removal 
Effectiveness  Performance 


AEA  IPT 
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Perform  Cause  and  Effect 
Analysis  (CAR) 


AEA  IPT 
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ONE  STOP  SHOP  FOR 
QUANTITATIVE  DEFECT  MANAGEMENT 


AEA  IPT  DRE  Dashboard 

Navigator 

Dashboard 


Organizational 


Objectives 


Planning  Phase 


Control  Charts: 


Process  Attribute 


Defects/Hr 


Defects/Unit  Size 


Hrs./Unit  Size 


Requirements 

Phase 


j 


■ 


Design 

Phase 


j 


Code  &  UT 
Phase 


1 


Requirements 


Design  Review 


Coding  &  Unit 


AEA  IPT 


Integration 


System  Testing 


DT/OT 
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Selected  Lean  Six  Sigma  Tools  for 
Quantitative  Defect  Management  Process 


•  Control  Charts 

•  Pareto  Chart 

•  Histogram 

•  Ishikawa  ("Fish  Bone") 

•  Five  Whys 

•  Process  Mapping 


i 


b|  imria  -il  !■  I 


M 

ri 


Together,  Lean  Six  Sigma  and  CMMI  help  AEA  I PT 
improve  performance  and  achieve  objectives  faster 


J 
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I  m prove  Performance  with 
Quantitative  Requirements  Management 


DoD-NAVAl  R  Directed  Continuous  Process  I  mprovement  (CPI )  by 
I  ntegrating  Lean  Six  Sigma  in  to  AEA I  FT  critical  processes 


w 
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Optimal  Process  Model 


Requirements  Management  Process  Model 


Elicitation 


Identify  Stakeholders 
Conduct  Fact  Finding 
Capture  Candidate  Technical  and 
Non-Technical  Requirements 


A  CUMULATIVE 
m  REQUIREMENTS 
GROWTH 

Progress  Through  Steps 


Analysis 


Categorize  and  Prioritize 
Establish  Database  Attributes 
Establish  Traceability 
Reconcile  and  Capture 
Decision  Rational 


Is,  Benchmarks,  Prototypes 


Assess 

Verify  Traceability 
Facilitate  Agreement 
Resolve  Deficiencies 
Establish  a  Baseline 

Verification 


•  Transform  to 

Engineering  Artifact 

•  Formalize  Traceability 

•  Place  under  CM 


Formalization 
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GAO  Reported 
Acquisition  Concerns 


•  Unsettled  requirements  in  acquisition  programs  can  create 
significant  turbulence 

•  Sixty-three  percent  of  the  programs  we  received  data 
from  (72  programs)  had  requirement  changes  after 
system  development  began 

•  These  programs  encountered  cost  increases  of  72 
percent,  while  costs  grew  by  11  percent  among  those 
programs  that  did  not  change  requirements 


AEA  IPT 
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NAVAI R  LESSONS  LEARNED 


•  Engineers  tend  to  resist  documenting  traceable 
requirements 

■  I  inability  to  trace  requirements  back  to  customer's  /  sponsor's 
requirements 

■  Requirements  creep  -  adding  requirements  not  necessary  to  meet 
user's  /  customer's  desires 


•  Lack  of  concurrence  among  the  stakeholders  of  the 
requirements  (collaboration) 

■  Key  contributor  to  requirements  instability,  which  leads  to  cost 
and  schedule  problems 


•  Lack  of  requirements  volatility  measures  (metrics) 


AEA  IPT 


UNCLASSIFIED  II  FOR  PUBLIC  RELEASE 


39 


NAVAI R  LESSONS  LEARNED 


•  Tendency  to  begin  preliminary  design  before 
requirements  are  verified  and  validated: 

■  Can  result  in  extensive  rework 

■  I  mpacts  accuracy  of  cost  and  schedule  estimates 


•  Resistance  to  having  a  Requirements  Change 
Control  Board  early  in  the  requirements  phase 


•  Requirements  too  loose/  broadly  written, 
complicating  requirements  decomposition 


•  I  nsufficient  time  dedicated  to  Requirements  Phase 


N) 


A.  J 
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AEA I  FT  Strategy  to 
Manage  Requirements  Volatility 


•  Stabilize  Requirements  Development  Process 

■  I  mprove  estimation  of  effort  to  develop  SRS  and  ensure  the  SRS 
is  completed  and  ready  for  design 

■  Control  and  I  mprove  the  Quality  of  Requirements  Specification 

•  Stabilize  Requirements  Management  Process 

■  I  nstitutionalize  the  Requirements  Change  process 

•  Develop  Quantitative  Requirements  Management 
(QRM)  Measures  for  a  Requirement  Volatility 
Index  (RVI): 

■  By  using  NAVAI R  Lean  Six  Sigma  initiatives 

■  Provide  a  CM  Ml  Level  4  and  Level  5  Framework 
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Quantitative  Requirements 
Management  -  7  Step  Process 
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Quality  is  never  an  accident,  it  is  always  the 
result  of  high  intention,  intelligent  direction  and 
skillful  execution;  it  represents  the  wise  choice 
of  many  alternatives. " 

William  A.  Foster 
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Questions? 


•  AEA I PT  LEAD 

•  AEA  I  PT  CHI  EF  ENGI NEER 

•  AEA  I  PT  PROCESS  I  IMPROVEMENT 
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I  nformation  Sources 


•  I  mprove  Quality  Performance 

■  Raja  Anantharaman,  Applied  Process  Solution 

•  Defect  Prevention 

■  David  LongStreet,  Softwaremetrics.com 

•  I  ncorporating  Quality  Throughout  the  Lifecycle 

■  Betty  Schaar,  BenchmarkQA 

•  Advancing  Defect  Containment  to  Quantitative  Defect 
Management 

■  Alison  A.  Frost  and  Michael  J .  Campo,  Raytheon 

•  NAVAIR's  Software  Engineering  Policies  and  Processes 

■  Barbara  Williams  ,  NAVAI R 
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